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1. Overview 



Since the early 1960s, federal workforce development programs have been 
characterized by separate funding "silos," distinct target populations, prescribed services, 
but very little in the way of accountability. Since the early 1980s, however, these 
programs have made considerable progress towards measuring and managing 
performance, even though such efforts have remained largely program-specific. Texas 
has emerged as one of the national leaders on workforce development performance 
management, especially since the passage of Senate Bill (SB) 642 in 1993 and House Bill 
(HB) 1863 in 1995. Among other actions, the earlier legislation created the Texas 
Council on Workforce and Economic Competitiveness (TCWEC) to serve as an 
overarching human resource investment council, and it encouraged greater coordination 
in planning, service delivery and performance management at both the state and local 
levels. 

HB 1863 went much further, mandating the creation of the Texas Workforce 
Commission (TWC) with operational responsibility for two dozen programs to be 
consolidated under it, among them the Job Training Partnership Act (JTPA), the Job 
Opportunities and Basic Skills (JOBS) and Food Stamp Employment and Training (Food 
Stamp E&T) program, and both the Employment Service (ES) and Unemployment 
Insurance (UI) benefit programs. Moreover, far more explicitly than any previous 
workforce legislation, state or federal, it defined the two major customers of workforce 
development services: first and foremost, employers; and (potential and incumbent) 
workers. HB 1863 also reinforced the systems evaluation emphasis which was initiated 
under SB 642.' 

TCWEC recently contracted with the Center for the Study of Human Resources to 
conduct three major tasks to assist it in performing its legislatively mandated 
responsibilities; 

1 . To assess current evaluation efforts for Texas workforce development services; 



' Trott and Baj (1996) provide an excellent review of the difference between program-specific performance 
management and a more systems-oriented approach. 



2. To develop a systems framework for evaluating Texas workforce development 
services; and 

3. To prepare an evaluation action plan detailing the steps required to get Texas 
from its current position to systems evaluation approach. 

This paper offers an initial assessment of Texas’ workforce development 
evaluation and performance management efforts. Section II clarifies important 
distinctions between performance management and evaluation responsibilities and 
between program-and systems-oriented evaluations. Section III identifies the key 
agencies and programs and their roles and responsibilities in Texas workforce 
development, including those for performance management and evaluation. Section IV 
then briefly describes major current and planned evaluation activities — including those 
involving data collection and reporting and ad hoc evaluation efforts — for the major 
Texas workforce development system actors. Section V synthesizes the assessment 
findings and presents a number of important gaps identified in the current and planned 
efforts and raises several key evaluation issues which need to be addressed. Section VI 
summarizes the next steps in this process. A bibliography follows. 

2. The Relationship Between Performance Management and 
Evaluation 

Performance management and evaluation are generally understood to be distinct 
but related accountability functions that have different purposes and require different 
types of analysis and data to support them. Typical differences between these functions 
are shown in Table 1. The emphasis on performance management and evaluation to 
promote public sector accountability has often come in the form of Congressional 
mandates for specific programs to perform periodic evaluations. Sometimes funding has 
been tied to compliance with performance measures and standards. Recently, with the 
passage of the Government Performance and Results Act of 1993, Congress mandated 
that all federal agencies develop outcome-based goals, measure their performance and 
report on their progress. 
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Table 1: Performance Management vs. Evaluation 





Performance Management 


Evaluation 


Purpose 


To determine and ensure near-term 
compliance with administrative 
and program goals and 
objectives regarding program 
outcomes. 


To determine the extent to which a 
program is achieving its broader 
legislative intent in terms of 
producing the effects on services, 
participants, etc. 


Definition 


an ongoing management process 
comprised of goals and 
objectives, performance 
measures and standards, and 
methods of establishing and 
adjusting standards, rewards and 
sanctions (Bamow 1992). 


an assessment of program 

implementation, outcomes and/or 
impacts on participants and other 
affected groups, as well as 
associated benefits and costs, often 
judged against broader legislative 
intent 


Information Produced 


extent to which programs comply 
with performance standards in 
the way the program is being 
implemented services being 
provided and the near-term 
outcomes experienced. 


• relationships among program 
components and services 

• relationships among participant 
characteristics, service 
interventions, and longer-term 
outcomes 

• relationships among services, 
impacts and benefits/costs 


General Question 
Addressed 


What is the association between the 
stated program goals/objectives, 
processes and near-term (gross) 
outcomes? 


What are the effects and net impacts of 
the program, judged in large part 
against its broader legislative intent? 


Key Questions Addressed 


1. Are funds being spent as 
intended? 

2. Are the intended target groups 
being served? 

3 . Are the intended services being 
provided? 

4. Are near-term outcomes being 
measured and reported as 
intended? 

5. Are participant outcomes and 
costs in compliance with 
established standards? 


1. What are the scope and range of 
outcomes/impacts? 

2. Were the intended outcomes and/ 
impacts achieved for participants 
served? 

3 . Did program participation yield net 
impacts as intended? For which 
groups? 

4. Did participation produce benefits 
in excess of costs? For whom (i.e., 
participants, taxpayers, society)? 



Performance management in employment and training programs usually refers to 
a system whereby programs and their outcomes are systematically judged against 
specified objectives (Bamow 1992). A performance management system includes 
performance measures, a method of setting standards for those measures, and rewards 
and sanctions based on performance relative to the standards. The measures tell how 
well the entities being judged are performing relative to what outcomes are expected of 
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the program. For each measure, a specific standard of acceptable performance must be 
set based on pure outcomes, outcomes of similar programs, or on statutory requirements. 
The rewards and sanctions can be in the form of funding changes and they should apply 
to programs depending on whether or not they meet the specified standards. A 
performance management system generally monitors processes and simple program 
outcomes. 

On the other hand, evaluations of employment and training programs have 
focused more on impacts rather than simple outcomes. Evaluation is concerned more 
with the broader legislative intent of the program in terms of producing expected impacts 
on client’s employment and earnings. Therefore, while performance management is an 
ongoing feature that continuously provides feedback to the managers and agencies 
responsible for the programs, evaluation can be conducted on*an occasional or one-shot 
basis. Evaluations often require multiple years of pre- and post-program information to 
gauge program impacts, while performance management systems must rely on shorter 
post-program periods in order to provide relatively quick feedback to program managers. 
Evaluations are generally more costly and intrusive and often require the use of 
comparison or control groups to identify what the outcomes for clients would have been 
without the program. Performance management systems are generally less intrusive, but 
they are not able to include impact measures. Evaluation analyzes the impact of a 
program, while performance management focuses on accountability in terms of the 
relationship of outcomes and processes to the stated goals of the program. 

Performance management allows an assessment of a program’s effects by 
monitoring selected outcomes against carefully selected standards. However, it may 
provide a skewed view of a program’s value because compliance with standards doesn’t 
necessarily indicate anything about relationships among a more extensive set of potential 
outcomes and the program’s impact on them. For example, evidence of compliance 
doesn’t necessarily provide good information about the relationship between outcomes 
and the organization of the program, or between outcomes and the way services are 
delivered to the client. Most significantly, the level of compliance does not inform us 
about the program’s role in these outcomes, i.e., was compliance responsible for the 
outcomes, or were these outcomes due to chance or other influences (Blalock 1990)? 
Ideally, short-term workforce performance measures would be highly correlated with 
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long-term impacts so that system managers would have confidence that short-term results 
were leading toward long-term impacts for customers. Performance management also 
encompasses other facets of accountability including fiscal compliance and audits, among 
others. 



3. Systems and Program Evaluation Approaches 

For the most part workforce development has existed as a collection of programs 
each addressing some population deemed to be in need of a particular type of 
employment and training service. Although individuals often qualify for services from 
multiple programs, the connections between programs have not been well articulated 
since programs are managed almost completely independently of each other. These types 
of employment and training programs have operated as functional silos, with funds and 
regulations flowing down from the top and communication and information sharing 
taking place internally. Any coordination efforts have usually focused on the bottom of 
the silos where actual client services are delivered, without strong coordination of policy 
decisions at the top. In this atmosphere, programs have tended to focus on rules, 
regulations, procedures, and requirements of a program, leading to an internal 
organization which focuses on functions such as fiscal operations, performance 
monitoring and others. The compliance monitoring role becomes primary, and minimal 
cross-program interaction takes place. Monitoring-for-compliance tends to take 
precedence over customer focus, and a sense of overall program purpose is not well 
developed (Trott and Baj 1996). 

A systems orientation is fundamentally different. Achieving the system’s purpose 
becomes the driving force behind programs and processes. The ultimate employment and 
earnings successes of clients and the competitiveness of companies reliant on the system 
become the new organizational focus. Management of the system takes on the role of 
developing and maintaining policies in common across programs. Management of the 
system also requires a well defined performance management and evaluation system that 
directs individual programs to achieve system goals. A performance management 
scheme then establishes an accountability framework that translates system goals and 
directives into performance measures and standards, a monitoring and reporting strategy. 
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and an evaluation strategy for the development and improvement of entire system (Trott 
andBaj 1996). 

4. Key Texas Agencies and their Roles in Workforce 
Development 

Texas has taken major steps to reform the way workforce development services 
are planned and delivered. Senate Bill 642 passed in 1993 and amended in 1995 by 
House Bill 1863 sets the framework for a single, integrated workforce delivery system to 
replace long-standing categorical programs. There are still many distinct federal and 
federal/state workforce programs, creating a framework Texas must work within and 
around. 

This legislation consolidates some 28 separate job training and employment 
programs in the newly created Texas Workforce Commission (TWC), authorizes the 
creation of local workforce development boards and directs the TWC to formula allocate 
funds for the larger job training and employment programs to local areas having certified 
boards and approved plans in place. In addition, a number of state-level advisory groups 
mandated by federal and/or state law are consolidated in the TCWEC.’ The new Council 
advises the governor and carries out the federal and state-required functions of each of 
the heretofore separate advisory councils. As such, it meets the requirements of a state 
human resource investment council set out in Title VII of the federal Job Training 
Reform Amendments of 1992. 

In Texas, TCWEC is the umbrella entity charged with planning, overseeing and 
evaluating all workforce development services in the state. In addition to many other 
responsibilities, it develops and recommends to the governor a strategic plan that serves 
as the framework for the budgeting and operation of all workforce development agencies. 
It is also responsible for evaluating the effectiveness of all workforce development 
services delivered in the state. 

The state workforce agencies function as administrative entities, developing rules 
consistent with Council policy and administering programs to achieve the state’s 
workforce objectives consistent with the performance expectations recommended by the 

’ Councils consolidated include: the State Job Training Coordinating Council, the Texas Council on 
Vocational Education, the Texas Literacy Council, and the Apprenticeship and Training Advisory 
Committee. 
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Council and approved by the Governor. With the exception of the Texas Rehabilitation 
Commission/ all of the mainline workforce development agencies are represented on 
TCWEC. The key agencies are depicted in an organizational chart of the Texas 
workforce development system (Figure 1). 



Figure 1 

Key Agencies in Texas’ Workforce Development System 




Under this arrangement, while the administering state agencies are presently more 
concerned with issues of performance management, TCWEC is primarily focusing on 
evaluating the effects and impacts of workforce services. 

It is important to note that constitutionally Texas is a “weak-govemor” state. The 
legislature is strong, and the governor does not have a cabinet. Executive power is 
scattered among some 200 boards and commissions — most of whose members are 
appointed by the governor — who balance differing executive as well as legislative 
priorities. The Council and the governor must work closely with the state legislature, and 



’ Rehabilitation agencies are exempted from the human resource council provisions under the 1992 JTPA 
Reform Amendments as well as under Texas’ wodcforce legislatioa 
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its powerful Legislative Budget Board (LBB) to develop goals and set expectations that 
will be enforced through state budgetary decisions. Table 2 summarizes the performance 
management and evaluation responsibilities placed on these agencies by major state and 
federal workforce and related legislation. 

Table 2: Performance Management and Evaluation Responsibilities from 
Major State and Federal Workforce and Related Legislation 



Agency 


Performance Management Responsibilities 


Legislative Budget 
Board 


• through its legislative ^propriations process, it requires state agencies 
to develop five-year strategic plans for monitoring performance against 
clearly defined outcomes. 


State Comptroller 


• conduct a management study to review the programs to be transferred to 
TWC [HB 1863] 

• review the Texas Department of Commerce’s Smart Jobs Fund program 
and TEA’S Adult Education program [HB 1863] 


TCWEC 


• conduct an occupation-specific analysis of job placement performance of 
each workforce education program [SB 642, as amended by HB 1863] 

• serve as state human resource council to advise the Governor on the 
development and implementation of state and local standards and 
measures relating to applicable federal and state human resource 
programs and coordination of such standards and measures [JTPA 
Reform Amendments of 1992] 


TWC 


• prepare and submit an annual agency performance report to the 
legislature, the Commission and the Council [HB 1863] 

• assess JTPA programs using the U.S. Department of Labor’s (USDOL) 
national performaivce standards [JTPA as amended by the Job Training 
Reform Amendments of 1992] 


HECB 


• develop and inq>lement a statewide system of core standards and 
measures of pe^ormance for post-secondary vocational education 
programs [C^l D. Perkins Vocational Education Act as amended by the 
Carl D. Perkins Vocational and Applied Technology Education Act 
Amendments of 1990] 


TEA 


• develop and implement a statewide system of core standards and 
measures of performance for secondary vocational education programs 
[Carl D. Perkins Vocational Education Act as amerrded by the Carl D. 
Perkins Vocational and Applied Teclmology Education Act Amendments 
of 1990] 


Local Workforce Boards 


• plan and oversee all area workforce training and services [SB 642, as 
amended by HB 1863] 
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Table 2 (cont) 



Agency 


Evaluation Responsibilities 


Legislative Budget 
Board 


• work closely with the Governor and TCWEC to develop goals and set 
expectations that will be enforced through state budgetary decisions. 


State Auditors Office 


• review each agency’s peifonnance against their established measiues 


State Comptrollers 
Office 


• review state agency peiformance and make recommendations to the 
legislature for improving the efficiency and effectiveness of state 
government (Texas Peiformance Review/ 


Legislative Oversight 
Committee 


• monitor the implementation and efficiency of the workforce development 
system [HB 1863] 


TCWEC 


• develop a strategic plan including goals, objectives and peiformarx^e 
measures for all programs [SB 642, as amended by HB 1863] 

• monitor the operation of the state’s workforce development programs to 
assess the degree to which the programs are effective in achieving state 
and local goals and objectives [SB 642, as amended by HB 1863] 

• evaluate the effectiveness of all workforce development programs [SB 
642, as amended by HB 1863] 

• assess the effectiveness of the major workforce development programs 
against the core measures in the state strategic plan [Memorandum of 
Understanding between TWC and TCWEC] 

• monitor the implementation and evaluate the effectiveness of the strategic 
plan [JTPA Reform Amendments of 1992] 


HECB 


• conduct annual evaluation to address any barriers resulting in lower rates 
of access to [postsecondaiy] vocational education programs and evaluate 
the progress of individuals who are members of special populations [Carl 
D. Perkins Vocational and Applied Technology Mucation Act 
Amendments of 1990] 


TEA 


• annually evaluate vocational education programs to address any barriers 
resulting in lower rates of access and evaluate the progress of individuals 
who are members of special populations [Carl D. Perkins Vocational and 
Applied Technology Mucation Act Amendments of 1990] 

• evaluate effectiveness of all state administered adult education programs 
and services [Adult Education Act, as amended by the Hawkins-Stafford 
Elementary and Secondary School Improvement Amendments of 1988] 


Local Workforce Boards 


• plan and oversee all workforce training and services and evaluate all 
workforce development programs in the area [SB 642, as amended by HB 
1863] 

• monitor and evaluate the effectiveness of career development centers, 
state agencies, and other contractors providing workforce and training 
services to ensure performance is consistent with state and local goals and 
objectives [SB 642, as amended by HB 1863] 



Vor a recent example, see: Office of the State Comptroller (1995). 
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5. Current and Planned Texas Workforce Evaluation 
Efforts 

This section examines evaluation efforts by workforce development agencies and 
programs in Texas, beginning with TCWEC and its work on core performance measures, 
One-Stop Career Centers and related areas: in the existing legislative context, TCWEC 
has primary evaluation responsibility for the workforce system. It also looks at program- 
specific activities within major agencies where these activities are noteworthy. Recent ad 
hoc evaluation efforts are referenced as well where appropriate. For many programs, 
once TCWEC's work has been described, there is simply little to note: in the State's Food 
Stamp E&T, School-to-Work,^ Skills Development Fund (TWC), Smart Jobs Funds 
(TDoC), and the host of smaller workforce development programs, beyond compliance 
monitoring, there has been no history of outcomes-based performance management or 
evaluation. 

5.1. TCWEC 

Under SB 642 and now HB 1863, TCWEC shoulders major responsibility for 
evaluating the Texas workforce development system, as well as for advising the 
Governor and operating agencies on key dimensions of performance management for the 
State's workforce development services. Its activities in performance management and 
evaluation since 1993 have encompassed the mandated federal roles (Table 2 above), as 
well as major systems evaluation efforts, in part stemming from its participation as one of 
six states selected for the National Governors' Association's Performance Management 
Project,* and in part from its work to develop a more systemic evaluation approach for the 
State's One-Stop Career Centers grant. 



*To date, the State has had little role to play in School-to-Woik in that program grants have been local in 
nature; Texas has a state grant application pending with the National School-to-Woik Office which would 
feature TCWEC in the evaluative role. 

^Funding for the state's participation in the NGA-sponsored Performance Management Project only 
extended for 1 8 months; Texas' work on core measures preceded this effort and has continued after its 
conclusioa 
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5.1.1. Core Performance Measures’ 



In the spring of 1994, TCWEC staff developed a set of eight (8) core measures as 
part of their responsibilities under SB 642 for tracking the performance of all workforce 
programs and conducting an annual, comprehensive evaluation of the workforce system. 
This effort was linked to the system and program goals developed by the Council for the 
Texas workforce system (Table 3). The eight core measures were adopted in concept by 
the Council and approved officially by then-Govemor Ann Richards in June 1994. Two 
additional measures were targeted for further exploration as potential measures for future 
adoption: training-relatedness of job placements, and cost-effectiveness. 

Table 3: System and Program Goals and Selected Objectives for 
the Texas Workforce Development System 



System Goals 

Goal #1 — To develop a statewide system supporting local career development centers where all workers, 
clients and employers can conveniently access a network of information and services responsive to 
their individual needs. 

Goal #2 — To develop a state/local strategic planning, evaluation, and accountability system for the state's 
woridbrce development program and activities. 

Program Goals 

Goal #3 — ^All Texans will have the literacy, basic education and basic workplace skills necessary for 
educational and career advancement. 

Goal #4 — Participants/workers will acquire the occupational skills to meet workplace requirements for 
long-term employment and work toward sustaining employment in high-skill, high-wage occupational 
areas. 

Goal #5 — ^All youth will be prepared with the knowledge, skills, and behaviors necessary to make the 
transition into meaning^, challenging and productive pathways in high-skill, high-wage careers, and 
for life-long learning. 



An Interagency Core Performance Measures Workgroup — with active 
representation from TDoC, TDHS, TEA, TEC, the HECB, local boards and others — ^was 
then convened by TCWEC to operationalize definitions for these measures and to 
determine the requisite core data elements to support them. In June 1995, operational 
definitions for five (5) of the core measures were adopted by the Council and officially 
approved for implementation by Governor George Bush. These measures — three labor 
market measures, one educational achievement and one access/equity measure (Table 

’This section draws on numerous internal memoranda, quarterly Council briefing and action items and 
conversations with current and former TCWEC staff. 
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4) — were subsequently incorporated into SOICC's ongoing follow-up system in July 
1995.* Job placement training-relatedness is also being tracked for those programs 
offering vocational-specific training, but not as a core measure. 



Table 4: Core Performance Measures for the Texas Workforce System 



Operational Measures 

Labor Market Outcomes 
L Entered E]i^)Ioyment Rate 

2. Earnings Gain Rates 

a Earnings Gain Rate, Based on Previous Earnings 

b. Earnings Gain Rate, Based on Program Entry 

c. Average Earnings Gains Based on Previous Earnings 

d. Average Earnings Gains Based on Program Entry 

3 . Employment Retention Rate 

Learning Outcomes Access/Equity 

4. Educational Achievement Rate 

5. Access/Equity Measures 

a Participation Equity Rate 

b. Target Population Successful Outcome Rates 

c. Target Population Group Identifiers For All Programs (gender, date of birth, race/ethnic 
groups, disabled individual) 

d. Target Population Group Identifiers To Be Reported On If Currently Collected By Program 



Measures to be Operationalized 

6. Skill Attainment 

7. Program Advancement 

8. Customer Satisfaction 

a Employer Satisfaction 
b . Employ ee/Clie nt Satisfactio n 



Measures for Future Consideration 

9. Training-Relatedness of Placement 

10. Program Cost Effectiveness 



®More detail on the SOICC follow-up effort is provided below; see Froeschle and Anderberg (1995). Note that 
SOICC was initially created as an independent agency under federal vocational education legislation in the 
midl970s. SB 642 then made SOICC an administrative division of TCWEC. Most recently, HB 1863 moved it to 
TWC. 




12 18 



SOICC's Automated Student and Adult Learner Follow-Up system, which is 
financed primarily with an array of "soft" federal grants, began tracking labor market 
outcomes and subsequent educational progress for many Texas community and technical 
education students in 1988. It now encompasses most major Texas workforce 
programs — JTPA, Employment Services, JOBS, Food Stamp E&T, publicly funded 
secondary and postsecondary Vocational and Technical Education, and a single Adult 
Education cooperative — ^providing employment, earnings and educational outcomes data 
for the relevant student/leamer exit cohorts at intervals of six months, one year and 
annually thereafter. These data are central to TCWEC's role in evaluating the Texas 
workforce system's performance over time. 

Preliminary research has been done on the three nonoperational, core measures — 
skill attainment, program advancement, and customer satisfaction — as well as on the two 
additional noncore measures which were identified for further consideration — ^training 
relatedness and cost-effectiveness. However, much work still remains. The last three 
core measures present very different challenges from the earlier set. There is no 
consistent, reliable and inexpensive data set available to serve as the basis for these 
measures, as (supplemented) UI wage records have for the labor market measures; 
instead, each may require identification or development of a standardized instrument or 
criteria for measurement. Some of this developmental work is being performed as part of 
TCWEC's contractual responsibilities under the State's One-Stop grant. 

5.2. One-Stop Career Centers Evaluation 

The One-Stop Career Centers effort has the potential to be the proving ground for 
a systems evaluation approach for the State's broader workforce development system. It 
embodies many of the characteristics and features of the workforce development system 
envisioned in SB 642 and HB 1 863, and it has moved deliberately towards adopting the 
five core measures, as well as pilot testing customer satisfaction measures for future use 
statewide and incorporating continuous improvement as a guiding principle of its 
performance management approach. TCWEC is also working with the One-Stop Centers 
to develop a strategy for gauging cost-effectiveness which may serve as the pilot for the 
Texas workforce development system. 
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TCWEC worked with the One-Stop Centers staff to develop, pilot and analyze the 
results of a customer exit survey in late 1995 and early 1996, a key step in instituting 
client-based customer satisfaction measurement. TCWEC and One-Stop staff have 
begun finalizing short-term progress indicators required by USDOL for the Centers as a 
condition of the grant. The progress indicators, include approved by the Performance 
Measurement Workgroup in May 1996: 

1 . UTILIZATION BY CUSTOMERS: reported number and type of customers 
served during the quarter. 

2. ENHANCED COMPREHENSIVENESS/ACCESSIBILITY TO 
INFORMATION/SERVICES: Enhancements to the accessibility and/or 
comprehensiveness of information/services provided during the quarter. 

3. ENHANCED CUSTOMER CHOICE: Enhancements to the information/services 
provided to customers that reflects customer needs and/or improves choice 
available to customers during the quarter. 

4. ENHANCED INTEGRATION: Enhancement to the integration of 
administrative, service delivery and or governance functions during the quarter. 

However, it has not been possible to incorporate One-Stop Career Center 
participants into SOiCC's follow-up system thus far. One-Stop participant records lack a 
"flag" which would allow them to be identified for tracking in the SOICC follow-up 
system. This is a serious omission, but one which can be corrected in the months to 
come. It clearly postpones conducting a more in-depth evaluation of the effects of One- 
Stop participation. 

5.3. Texas Skills Standards Board 

TCWEC also has statutoiy responsibility under HB 1863 for staffing the newly 
created Texas Skills Standards Board. Its activities have just recently begun in this role. 
Among other things, the Board intends to: 

• develop a framework for establishing valid and reliable skills standards; 

• facilitate and coordinate the development of such standards by industry groups; 
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• enter into agreements with other states for mutual recognition of credentials to 
enhance the portability of skills. 

• validate existing and/or developing national skills standards; and 

• ultimately, integrate skills standards into the broader workforce development 
system's performance evaluation and measurement approach. 

At this stage, it is too soon to tell where the Texas Skills Standards Board's work 
will take it. It held its first meeting in August 1996. However, since skill attainment is 
one of the core performance measures, the skill standards system is an important 
component of accountability for the workforce development system. The skill standards 
system will provide a means to measure whether publicly-funded and regulated programs 
are preparing workers with the skills needed to meet the requirements of the workplace. 

6. Related Performance Review Activities 

In addition to TCWEC, three other state entities are also involved in the review of 
workforce development services in Texas as part of performing their larger budget, 
performance review and oversight responsibilities. The Legislative Budget Board (LBB), 
through its biennial legislative appropriations request process requires state agencies to 
develop five-year strategic plans that include goals, objectives, strategies and measures 
for monitoring performance against clearly defined outcomes.’ “Statements of impact” or 
performance measures are required in four general areas; outcomes, outputs, efficiency 
and explanatory measures. 

Each agency’s strategic plan, including the key measures and its budget request, is 
updated every two years and negotiated with the LBB as part of the state budgeting 
process prior to each regular legislative session.'® It, of course, is then debated in the 
legislature as part of the biennial appropriations process and finally set in law. 



*For more on this process, see: [Texas] Governor’s Office of Budget and Planning/Legislative Budget 
Board (1995). 

'®Note that in the 1995 session, the eight core workforce performance measures were incorporated iitfo the 
Legislative Appropriations Request for the newly created Texas Workforce Commission, a move which 
may have been somewhat premature. At that time, only five of the eight measures had been formally 
adopted; none had actually been implemented or measurement systems established. The three remaining 
core measures are in varying stages of development even now. 
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The State Auditor's Office, an arm of the legislative branch, reviews each 
agency’s performance against their established measures. This information is used by the 
LBB, the governor’s office the state legislature and by the agencies themselves to 
influence future planning, resource allocation and operational decisions of state 
government. To the extent that clear outcome measures and the data systems required to 
support them are in place, assessments by the State Auditor’s Office represent 
evaluations of service impacts. Such measures, however, are not yet in place for the 
state’s workforce agencies or for the new workforce system. 

The State Comptroller is also actively involved in reviewing the performance of 
Texas agencies. Soon after coming into office. Comptroller John Sharp initiated an 
ongoing effort to review state agency performance and make recommendations to the 
legislature for improving the efficiency and effectiveness of state government, known as 
the Texas Performance Review (TPR)." While these are comprehensive reviews, they 
tend to focus on areas of state government where change is most needed. Once the issues 
are identified and addressed and changes made, TPR's focus shifts to other policy areas. 
Consequently, there is not an ongoing assessment of agency performance like that done 
by the State Auditor. Further, the Comptroller’s reviews clearly concentrate on agency 
performance rather than on impact. 

Since the early 1990s, the Comptroller's Office has given a great deal of attention 
to workforce development. Reviews of the various workforce agencies and the services 
they provide led to major recommendations for program consolidation and for creating a 
single, integrated delivery system similar to that now found in state law (HB 1 863). 

6.1. Texas Workforce Commission“ 

This characterization focuses only on TWC's major workforce programs (e.g., 
JTPA, JOBS, ES). It does not examine the many smaller ones which now round out 
TWC's workforce "toolkit", such as the Senior Texans Employment Program, Project 
Reintegration of Offenders (Project RIO) and others, most of which lack a full-fledged 



"For a recent example, see: Office of the State Comptroller (1995). 

'^This section is based on a review of documents and reports from the various programs. Also, it benefited 
considerably from lengthy discussions with many TWC staff, including Carolyn Young, Jim Gaston, 
Brenda Lovett, Tom Depalermo, Mike Fernandez, Emily Zimmet, Jere Goldgar and Juan Garcia. 
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performance management mechanism; nor have they been subjected to systematic 
evaluations. 



6.1.1. JTPA 

JTPA programs have long had the most comprehensive performance management 
system of any workforce program (Bamow and King 1996). Its system has had well 
articulated roles and responsibilities at the national, state and local levels, and its 
performance goals have been clearly stated and reliably measured on the whole since the 
early 1980s. Over time, there have been attempts to ensure that the near-term measures 
are valid indicators of longer-term net impacts on the goals of interest, especially 
increased employment and earnings.'^ 

TCWEC now has assumed the performance management role formerly played by 
the State Job Training Coordinating Council, advising the Governor on performance 
standards adjustments, incentives and sanctions. TCWEC also has responsibility for 
evaluating JTPA efforts in the state as part of its workforce systems evaluation, even 
while TWC has more operational performance management responsibility. 

JTPA has been the object of intense scrutiny (and controversy) in recent years 
following completion of a net impact evaluation of Title IIA programs for adults and 
youth. This evaluation, conducted by MDRC and Abt Associates (Orr et al. 1995) in 
sixteen sites around the nation, found modest statistically significant positive net earnings 
impacts for adult women, positive but insignificant impacts for adult men, and no impacts 
for youth. Other studies have examined long-term results (e.g., U.S. G.A.O., March 
1996). 

There have also been a number of ad hoc evaluations of Texas JTPA programs in 
recent years. Texas A&M University's Public Policy Resources Laboratory conducted an 
outcomes evaluation of Texas Dislocated Worker programs under Title IQ of JTPA 
(1993). In addition, TDoC's Workforce Development Division staff performed an 
outcomes analysis of JTPA Title HA programs (TDoC 1992). Finally, King et al. (1995) 
recently completed an analysis of Texas JTPA Title IIA longer-term employment and 
earnings (gross) outcomes (based on two years pre- and post-termination UI wage records 



'bindings from a validation study using the JTPA net impact data are forthcoming from the U.S. 
Department of Labor. The research team on this study is comprised of Dr. Burt Bamow of Johns Hopkins 
University's Institute for Policy Studies, and others. 



data), along with field interviews with program staff in selected "success stories" sites, 
for the National Commission for Employment Policy and the USDOL.'^ None of these 
efforts can be categorized as ongoing or systemic. 

6.1.2. JOBS 

Provisions in the prior national authorizing legislation (the Family Support Act of 
1988) required the Secretary of the U.S. Department of Health and Human Services 
(USHHS) to recommend a set of outcomes-based performance measures and related 
language by October 1993 (USHHS n.d.). USHHS failed to do so, however, finally 
coming forward in 1994 with a schedule which proposed implementation of national 
outcome measures by 1998. JOBS has continued to operate nationally with only program 
participation rates to guide them. It will be supplanted now by varying state welfare-to- 
work programs under the emerging framework of the recently signed welfare reform 
legislation, the Personal Responsibility and Work Opportunity Reconciliation Act of 
1996. 

Texas DHS adopted its own regional outcome goals (e.g., entered employment 
rates, target wage rates) for the State's JOBS programs. It also has participated actively 
in TCWEC's Core Measures process and contributed program "seed" records to SOICC's 
follow-up effort. Texas Legislative Appropriations Rider (#33) required external 
measures to be reported by TEA and TWC as of December 1995. TEA reported the 
count of all clients receiving a GED, and TWC reported six measures; 

• # of clients employed after job search 

• # of clients employed in the year 

• average wage for job search clients entering employment 

• average wage for all clients entering employment 

• total # of clients entering employment 

• # of clients who have completed the 12th grade. 



’ Vhis paper was part of a larger report produced for NCEP and USDOL analyzing Texas and niinois JTPA 
programs. The Illinois research was performed by Dr. Charles E. Trott and John Baj of Northern Illinois 
University's Center for Goverrunental Studies. 



Several large-scale JOBS program evaluations have been conducted. A team of 
researchers at the State University of New York-Albany, working in conjunction with 
Field Associates in nine states,” produced a series of reports on JOBS program 
implementation.'^ In addition, a multi-state, net impact evaluation of JOBS was initiated 
in the early 1990s by USHHS and is being performed by MDRC, in conjunction with 
researchers at Social Policy Research Associates and Child Trends, Inc. (e.g., Freedman 
and Friedlander 1995). 

The Texas JOBS Program Evaluation, a quasi-experimental net impact evaluation 
of the program conducted by the Center for the Study of Human Resources, is probably 
the closest thing to a comprehensive, systems-oriented evaluation that has been 
conducted of any Texas workforce development program. It evaluated near- and longer- 
term employment, earnings and welfare recidivism impacts, coupled with a benefit-cost 
analysis, for AFDC recipients participating in the Texas JOBS program in the early 
1990s.” This evaluation was initiated by the Center and developed and financed by the 
four agencies which constituted the primary collaborative partners in JOBS — DHS, 
TDoC, TEC and TEA. JOBS participants then received an array of what are now termed 
workforce training and education services from programs operated by each of these 
agencies as well as community and technical colleges. The evaluation found statistically 
significant, but modest, net impacts on employment, earnings and recidivism, which were 
primarily associated with participation in the program’s education and training 
components. 



6.1.3. Food Stamp Employment and Training 

Food Stamp E&T is primarily a process- not an outcomes-oriented program. Like 
JOBS, its principal performance requirements are associated with participation rates 
rather than entered employment and wage rates or retention. 

As reported by Puma and Bumstein (1994) of Abt Associates, Inc., the results of 
the nationwide Food Stamp E&T impact evaluation were not encouraging: participation 
in the program was not found to be associated with any positive net impacts on 

”Texas was one of the JOBS Implementation Study states. 

'Reports available from this study include; Lurie and Hagen (1993, 1994). The final JOBS 
In^ilementation Study report is ejqiected sometime this year. 

‘^Reports from this evaluation include: King et al. (1993); King et al. (1994); King and Schexnayder 
(1994); and Schexnayder and Olson (1995) 
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employment or earnings, and, thus, also failed to yield favorable benefit/cost results as 
well. In Texas, researchers at the Center for the Study of Human Resources have been 
conducting an evaluation of the State’s Food Stamp E&T/JOBS Program Conformance 
Demonstration since 1993.'* This evaluation, funded by the U.S. Department of 
Agriculture's Food and Nutrition Service via the Texas DHS (now TWC), is examining 
the benefits and costs of bringing the Food Stamp E&T program eligibility, activities and 
services more in conformance with the larger JOBS program for AFDC recipients. Final 
evaluation findings are due to be reported out in late fall 1996. 

6.1.4. Employment Service 

ES also has had some experience in performance management and evaluation. 
Nationally, ES was one of the first federal workforce programs to move into the 
outcomes-based performance management arena. But, after several initial forays — ^and 
despite relatively explicit language in the federal authorizing legislation (W agner-Peyser, 
Section 7b) allowing Governors to reward local offices for performance — such efforts 
have languished. Texas as a state has done more than most to bring expectations and 
practice for its ES programs into the performance management arena, along with JTPA, 
JOBS and others. 

Meyers (1991) conducted a quasi-experi mental net impact evaluation of a set of 
job search seminars developed within the Texas ES program which led to their 
subsequent expansion and a series of seminar refinements. 

6.2. Related TWC Data Collection 

TWC is currently reviewing the underlying federal/state data collection and 
reporting systems for the more than two dozen programs now under its purview to 
determine the most effective and efficient means of carrying out its continuing 
responsibilities for these efforts.” TWC is working closely with its private sector 



'*Reports from this evaluation are available from the Center, including O'Shea et al. (1995, 1996). The 
final reports from this evaluation will be completed in September. The Center is also performing a similar 
analysis for Hawaii's Food Stamp E&T/JOBS demonstratioiL Several other states also have been operating 
Conformance demonstrations. 

'*This brief discussion draws on interviews with Mike Fernandez, TWCs Director of Information Services. 
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partners to develop a new information system. In the near term, TWC is maintaining the 
old systems. 

6.2.1. SOICC 

SOICC is now part of TWC, however, it also provides data to numerous state 
workforce-related agencies. SOICC’s Automated Student and Adult Learner Follow-Up 
System examines public and higher education "seed" records through a process of record 
linkages which enables a matching of records from individuals in each of these systems. 
The primary focus of this follow-up system is the tracking of completers of career and 
technology education programs (Froeschle and Anderberg 1995). These “seed” records 
are then matched with UI wage records in order to assess the labor market outcomes for 
those students who exited high school or other programs. 

Follow-up reporting is utilized by a number of state agencies for performance 
management and evaluation efforts. TCWEC and TWC use data from this system for 
tracking the outcomes from programs under their direct control (e.g., JTPA, JOBS, Food 
Stamp E&T), as well as for required reporting to the LBB. In addition, the Comptroller 
has indicated an interest in using the data to conduct its performance reviews. The 
Coordinating Board has moved away from data collected at specific institutions to a 
reliance on the SOICC’s follow-up information for their outcomes-based program 
evaluations (i.e., institutional effectiveness). TEA may also be interested in employing 
the SOICC follow-up data for this same reason. Both TEA and the Coordinating Board 
currently examine SOICC follow-up data for evaluating the State’s Tech Prep programs. 

6.3. Texas Higher Education Coordinating Board 

There are two main program areas of interest here. The first is postsecondary 
Vocational and Technical Education, programs administered and overseen by the 
Community and Technical Colleges Division (CTCD) of the Texas Higher Education 
Coordinating Board. The second is Tech Prep. 

6.3.1. Postsecondary Vocational and Technical Education 

Current performance management and evaluation efforts for postsecondary 
vocational and technical education offer a very sharp contrast to those in place just a few 



years ago. Beginning in 1988, somewhat before the Carl D. Perkins Education and 
Technology Act Amendments of 1990 began pushing vocational education nationally 
towards an outcomes-based approach to accountability,” CTCD joined SOICC to begin 
tracking labor market outcomes for former vocational/technical education students. With 
the impetus of the Perkins Act and building on the performance measures established by 
the State's Committee of Practitioners, in 1993 the Assistant Commissioner for 
Community and Technical Colleges appointed a Task Force on Institutional 
Effectiveness, comprised of state and local community and technical college 
representatives, "to develop a state-level evaluation and continuous improvement plan 
focusing on workforce education and academic programs at community and technical 
colleges in Texas."’’ The Task Force's primary task was to review the disparate elements 
of the existing performance management efforts — e.g., accreditation by the Southern 
Association of Community Colleges, performance measurement from the Perkins Act, 
additional state mandates — together with the typically ignored fiscal accountability, and 
to integrate these into a single evaluation process. 

This institutional effectiveness process has three main components: 

1. Annual Self-Evaluations (now fully automated) performed by the individual 
colleges. 

2. Annual (state-required) Data Profiles, summarizing current performance data and 
annual progress toward state goals, as measured by six "critical success factors": 
mission; effective use of (human and fiscal) resources; access; achievement; 
quality of academic areas; and quality of technical program.” SOiCC-generated 
follow-up data provide vital information for these state and college Annual Data 
Profiles, especially regarding the achievement (labor market based) and quality of 
technical programs critical success factors.” 

3. On-site Peer Reviews, conducted every four years, using the first two data-based 
components. CTCD sends the results of these reviews to the college's president 



’“This description is based upon a review of CTCD documents, as well as lengthy discussions with Mollie 
Boyd, Ron Curry and Martha Obum of CTCD, Helen Geraitis, formerly of CTCD, and Marc Anderberg of 
SOICC. 

’’Texas Higher Education Coordinating Board (January 1996), p. 1. 

’’ibid, p. 4ff. 

’’Obtaining more stable and secure funding for SOiCC's follow-up effort, not surprisingly, was one of the 
major concerns expressed by CTCD staff. 
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for the college's response. The on-site reviews have just entered the fourth year of 
implementation.^^ 



At present, the Institutional Effectiveness process has just approached what 
CTCD refers to as the "closing the loop" phase: CTCD is currently reviewing college's 
recommendations (e.g., close program x) with their actual implementation. The 
institutional effectiveness results are shared with the presidents of the colleges, as well as 
the technical deans. 

At this point, CTCD views the core measures somewhat cautiously, in large part 
because they have already established a systems-oriented evaluation framework which 
embodies many of the features being considered by TCWEC as part of its legislative 
mandate. Parts of the core measures are incorporated into the institutional effectiveness 
process although not explicitly. CTCD also recognizes that documenting success on 
these core measures could assist them in portraying what Texas' community and technical 
colleges do well. 

6.3.2. Tech Prep 

The HECB awarded Perkins Act funds to the Region V Education Service Center 
(in Beaumont) to prepare a report on the tech prep initiative in Texas. The goal of this 
effort was to document the overall effectiveness of the tech-prep initiative in Texas 
during the last five years by consolidating previous evaluators’ recommendations and 
conducting original research (Brown 1996). Further, this report sought to identify key 
elements for assessment of tech-prep program effectiveness and to efficiently collect 
baseline data relative to these elements. A status report, published in April of 1996, 
provided a snapshot view of the development of 25 regional tech-prep consortia within 
the Governor’s 24 planning regions. An updated status report and preliminary impact 
results are expected as part of this effort in late 1996. 



^^Interestingly, accxsrding to CTCD, while the colleges rated fully 40 percent of their technical programs as 
"exemplary", the peer reviews only designated 4 percent as such. 
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6.4. TEA 



6.4.1. PEIMS and AEIS 

The Public Education Information Management System (PEEMS) and Academic 
Excellence Indicator System (AEIS) are two extensive databases maintained by TEA. 
Both data sets are quite detailed and designed to support numerous performance 
management and evaluation functions for the public education system in Texas. The 
PEIMS and AEIS databases include information for Texas public primary and secondary 
educational institutions including student performance and demographics, and staff and 
financial information among other items. 

PEIMS will be used by the Center in its evaluation of the Texas Welfare Waiver 
{Achieving Change for Texans), after which it will be clearer how appropriate the 
database is for conducting evaluation. PEIMS and AEIS are both being used to produce 
annual school performance reports. PEIMS provides the data for the annual school 
district reports for each district in Texas. The AEIS data supports the Texas School 
Performance Report for each individual school. Also, the Education Productivity 
Council is currently using the AEIS database for their evaluation of Texas public schools. 

6.4.2. Adult Basic Education and Literacy 

TCWEC has been working with TEA'S Adult Education and literacy programs to 
validate the Intake Assessment Interview which is intended to establish the baseline for 
evaluating adult education/literacy students' skill attainment. TCWEC staff have also 
coordinated with TEA on developing a framework for systematically measuring adult 
education/literacy skill attainment. To that end, TEA has taken the leadership in defining 
performance measures and developing an assessment system for determining student 
progress in adult education and literacy. TEA has designated the Adult Education 
Professional Development and Curriculum Consortium, a group of special projects, to 
design such a system. The system would include an initial assessment of the literacy 
proficiencies of adults, primarily those who plan to participate in workforce training 
programs. The assessment system will also include a baseline assessment and an 
assessment of client progress. 
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The intent of the Performance Measures Assessment System is that it be used 
throughout the workforce development system for clients who receive adult education 
and literacy services. The development of the assessment framework is currently funded 
through the TEA with the Adult Education Professional Development and Curriculum 
Consortium contractors, using federal National Literacy Act discretionary funds. 

7. Synthesis, Issues and Gaps 

7.1. Synthesis 

So, what can be said about the existing Texas workforce development evaluation 
system? Several findings and observations can be offered by way of synthesis. 

First, with some very notable exceptions, like most states across the country, 
Texas still features a relatively fragmented array of program-by-program performance 
management mechanisms. These are primarily driven by federal legislative and 
regulatory requirements. They should not be viewed as responses to broader workforce 
systems concerns. Moreover, in terms used by Trott and Baj (1996), policy coordination 
among TWC, TEA, Coordinating Board and other workforce related efforts, policy 
coordination is clearly lacking. Some of these programs key on labor market or other 
program outcomes, while others remain overly concerned about process. 

Second, to the extent that workforce-related evaluations have been conducted in 
Texas in recent years, they have been ad hoc rather than ongoing efforts. These 
evaluations have been process or implementation studies as often as they have been 
outcomes-oriented. Only one or two have involved experimental or quasi-experimental 
methods (i.e. attempting to gauge net impacts) or any type of benefit-cost or cost- 
effectiveness analysis. 

Third, Texas is beginning to move strongly and relatively deliberately towards a 
systems evaluation orientation for its workforce development system. Several elements 
of this movement constitute the notable exceptions referred to above, namely: 

• TCWEC's efforts to develop strategic systems and program goals, and their 
associated core performance measures for all workforce services; 

• The Coordinating Board's (CTCD) Institutional Effectiveness system for 
community and technical colleges; and 
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• SOICC's longstanding initiatives for tracking student/participant and employer 
outcomes in support of most of Texas' workforce efforts. 

These three efforts taken together provide a relatively solid base from which to 
continue the work towards a true systems orientation and an evaluation system to match. 

Fourth, while the Texas workforce development system has clearly identified 
employers as the primary customer for workforce services, very little in the current or 
planned performance management and evaluation initiatives even make reference to 
them. The Customer Satisfaction core measure is eventually slated to have a submeasure 
pertaining to employers' satisfaction. This does not appear to place sufficient emphasis 
on this important customer if HB 1863's legislative intent is to fully addressed. 

Fifth, the data collection and reporting systems which are required to support 
Texas' workforce management and evaluation efforts need serious attention on a number 
of fronts. Several of the more important concerns are: 

• Existing data collection and reporting mechanisms are largely driven by federal 
reporting requirements which tend to inconsistent as well as incompatible with 
real performance management and certainly evaluation under HB 1863; 

• Many of these data systems are in major transition as TWC continues to integrate 
the various programs under its umbrella; 

• SOICC has continued to perform a major support service to many, if not most, of 
the mainline workforce development programs, operating almost exclusively on 
'soft' (federal or other) grant funds developed on its own. A performance 
management and/or evaluation system cannot operate effectively if it fails to 
invest 'hard' administrative dollars in one of its more important foundations. 

7.2. Issues and Gaps 

Based on this preliminary assessment of workforce development performance 
management and evaluation efforts. Center researchers have also identified a number of 
issues and gaps which will need to be addressed in the coming months. These constitute 
a mix of technical adjustments and corrections; data collection and reporting problems; 
and methodological difficulties. Other issues may surface as discussions continue with 
evaluation experts and the broader state and local workforce development system. 
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The issues and gaps identified initially include the following: 

Broader Issues/Gaps 

• The current system and program goals (and their associated objectives) for 
workforce development are in need of clarification and refinement. For example, 
is the program goal continuous improvement, return-on-investment or 
performance against standards? 

• Considerable work remains to be done to define operationally the three remaining 
core measures: skill attainment, program advancement, and customer satisfaction, 
both for the employee/client and especially the employer. It is becoming clearer 
that cost-effectiveness — in all likelihood, retum-on-investment (ROI) — measures 
will be required to round out the existing set of core measures. Note that a 
number of technical corrections and minor modifications are needed as well in the 
five existing measures as presently defined operationally, especially for the 
access/equity and some of the earnings gain measures. 

• A control/influence disconnect is now imbedded in HB 1863: TCWEC is directed 
to strategically plan for, oversee and evaluate all Texas workforce development 
services, including those within TWC, TEA, TDoC and the Coordinating Board, 
just as are local workforce development boards; local boards can control those 
(largely second-chance) programs which are under TWC administration, but they 
can only influence the others. It should be noted that the programs they only 
influence may be ten to fifteen times larger in size than the ones they control. 
This is more a performance management than an evaluation issue, but it is a 
critical one as TCWEC and TWC promote and support local workforce 
development boards. 



Data-related Issues/Gaps 

• UI wage records and related sources have inherent limitations in terms of 
coverage, frequency, timeliness, etc. These gaps raise issues which should be 
more explicitly addressed. It should be noted that, in the near future, Texas will 
be maintaining sixteen quarters of UI wage data on line, making any number of 
evaluative concerns far easier and cheaper to examine. 

• As Texas moves farther towards a "seamless" workforce system (federal laws and 
regulations permitting), the feasibility of instituting a "unit-of-service" 
information system approach for participant and student records should be 
explored. At present, some of the most serious inter-program incompatibilities 
pertain to the activity/component/service categories. 
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• Interrelated cost and participant/student data files will need to be generated to 
facilitate cost-effectiveness and ROI analyses if such measures are to be feasible. 
Cost accounting systems will need to be reviewed and every effort made to bring 
them into compatibility as well. 

• The integrity, consistency and reliability of the system’s basic "seed" records are 
often questionable, and these records constitute the basis of the SOICC follow-up 
effort. 

Methodological Issues 

• Net impact versus gross outcome evaluation methodologies will need to be 
reviewed to determine which methods are feasible and appropriate for Texas' 
workforce evaluation effort. 

• New methods will need to be developed to accurately measure the workforce 
system's impact on employers. Very little research appears to have been done in 
this arena, either conceptually or practically. In part, TCWEC's efforts on behalf 
of the Texas Skills Standards Board may have a role to play here. 



7.3. Next Steps 

There are two major steps to follow. First, Center staff are working with TCWEC 
and consulting affected state and local entities to develop the systems evaluation 
framework and to build consensus around it. Second, the Center will construct an 
Evaluation Action Plan detailing the steps required to institute a systems approach rather 
than the program-by-program one now in place. 
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